System for improving the performance of information retrieval-type tasks by identifying the relations of constituentes

ABSTRACT

A system is utilized for determining a relationship between first and second textual inputs. The system identifies constituents in the first textual input, having predetermined characteristics indicative of usefulness in determining the relationship. The relationship is then determined based on the constituents identified. The constituents can be eliminated from the first textual input, weighted in the first textual input, or simply annotated in one of a variety of ways.

The present application is a divisional of and claims priority of U.S.patent application Ser. No. 09/366,200, filed Jun., 18, 1999, thecontent of which is hereby incorporated by reference in its entirety.

REFERENCE TO COPENDING APPLICATION

Reference is hereby made to co-pending U.S. patent application Ser. No.09/220,836 filed Dec. 24, 1998 entitled “SYSTEM FOR IMPROVING THEPERFORMANCE OF INFORMATION RETRIEVAL-TYPE TASKS BY IDENTIFYING CLAUSESHAVING PREDETERMINED CHARACTERISTICS” and assigned to the same assigneeas the present application.

INCORPORATION BY REFERENCE

The following patent applications are hereby fully incorporated byreference:

Co-pending U.S. patent application entitled “METHOD AND SYSTEM FORCOMPUTING SEMANTIC LOGICAL FORMS FROM SYNTACTIC TREES”, filed Jun. 28,1996 and assigned Ser. No. 08/674,610; and

U.S. patent application entitled “INFORMATION RETRIEVAL UTILIZINGSEMANTIC REPRESENTATION OF TEXT”, filed Mar. 7, 1997 and assigned Ser.No. 08/886,814, both of which have been assigned to the presentassignee.

BACKGROUND OF THE INVENTION

The present invention is directed to a system for determining arelationship (such as similarity in meaning) between two or more textualinputs. More specifically, the present invention is directed to a systemwhich performs improved information retrieval-type tasks by identifyingrelations of constituents of documents being searched.

The present invention is useful in a wide variety of applications, suchas many aspects of information retrieval including indexing, pre-queryand post-query processing, document similarity/clustering, documentsummarization, natural language understanding, etc. However, the presentinvention will be described primarily in the context of informationretrieval, for illustrative purposes only.

Generally, information retrieval is a process by which a user finds andretrieves information, relevant to the user, from a large store ofinformation. In performing information retrieval, it is important toretrieve all of the information a user needs (i.e., it is important tobe complete) and at the same time it is important to limit theirrelevant information that is retrieved for the user (i.e., it isimportant to be selective). These dimensions are often referred to interms of recall (completeness) and precision (selectivity). In manyinformation retrieval systems, it is important to achieve goodperformance across both the recall and precision dimensions.

In some current retrieval systems, the amount of information that can bequeried and searched is very large. For example, some informationretrieval systems are set up to search information on a global computernetwork (such as the Internet), digital video discs, and other computerdata bases in general. The information retrieval systems are typicallyembodied as, for example, Internet search engines and library catalogsearch engines. Further, even within the operating system of aconventional desktop computer, certain types of information retrievalmechanisms are provided. For example, some operating systems provide atool by which a user can search all files on a given data base or on acomputer system based upon certain terms input by the user.

Many information retrieval techniques are known. A user input query insuch techniques is typically presented as either an explicit usergenerated query, or an implicit query, such as when a user requestsdocuments which are similar to a set of existing documents. Typicalinformation retrieval systems search documents in a larger data store ateither a single word level, or at a term level. Each of the documents isassigned a relevance (or similarity) score, and the informationretrieval system presents a certain subset of the documents searched tothe user, (typically that subset which has a relevance score whichexceeds a given threshold).

The rather poor precision of conventional statistical search enginesstems from their assumption that words are independent variables (i.e.,words in any textual passage occur independently of each other).Independence in this context means that a conditional probability of anyone word appearing in a document given the presence of another wordtherein is always zero (i.e., a document simply contains an unstructuredcollection of words or simply put “a bag of words”).

As one can readily appreciate, this assumption, with respect to anylanguage, is grossly erroneous. Words that appear in a textual passageare simply not independent of each other. Rather, they are highlyinter-dependent.

Keyword based search engines totally ignore this fine-grained linguisticstructure. For example, consider an illustrative query expressed innatural language: “How many hearts does an octopus have?” A statisticalsearch engine, operating on content words “hearts” and “octopus”, ormorphological stems thereof, might likely return or direct a user to astored document that contains a recipe that has as its ingredients andhence its content words: “artichoke hearts, squid, onion and octopus”.This engine, given matches in the two content words, may determine,based on statistical measures, that this document is an excellent match.In reality, the document is quite irrelevant to the query.

The art also teaches various approaches for extracting elements ofsyntactic phrases which are indexed as terms in a conventionalstatistical vector-space model. One example of such an approach istaught in J. L. Fagan, “Experiments in Automatic Phrase Indexing forDocument Retrieval: A Comparison of Syntactic and Non-SyntacticMethods”, Ph.D. Thesis, Cornell University, 1988, pp. 1-261. Anothersuch syntactic based approach is described, in the context of usingnatural language processing for selecting appropriate terms forinclusion within search queries, in T. Strzalkowski, “Natural LanguageInformation Retrieval: Tipster-2 Final Report”, Proceedings of Advancesin Text Processing: Tipster Program Phase 2, Darpa, 6-8 May 1996, TysonsCorners, Va., pp. 143-148; and T. Strzalkowski, “Natural LanguageInformation Retrieval”, Information Processing and Management, Vol. 31,No. 3, 1995, pp. 397-417. A further syntactic-based approach of thissort is described in B. Katz, “Annotating the World Wide Web UsingNatural Language”, Conference Proceedings of R.I.A.O. 97,Computer-Assisted Information Search on Internet, McGill University,Quebec, Canada, 25-27 Jun. 1997, Vol. 1, pp., 135-155.

These syntactic approaches have yielded lackluster improvements, or havenot been feasible to implement in natural language processing systemsavailable at the time. Therefore, the field has moved away fromattempting to directly improve the precision and recall associated withthe results of a query, to improvements in the user interface.

Another problem is also prevalent in some information retrieval systems.For example, where documents are indexed, such as in a typicalstatistical search engine, the index can be very large, depending uponthe content set, and number of documents to be indexed. Large indicesnot only present storage capacity problems, but can also increase theamount of time required to execute a query against the index.

The term “grammatical relations” is used to denote subject, object, andother constituents that can be identified on the basis of a syntacticanalysis. Linguists recognize that grammatical relations are not all ofequal status. For example, Keenan and Comrie have developed a summary ofhow different languages mark positions in a domain of relativizationwhich are assumed by noun phrases. The summary is referred to as theAccessibility Hierarchy (or hierarchy of accessibility) and is describedas follows.

Topic (optional)>subject>direct object>indirect object>object ofpreposition or post position>genitive (possessor)>object of comparison.

The hierarchy of accessibility illustrates a generalization that thelower a noun phrase is on the hierarchy, the less likely it is that thisnoun phrase will be expressed by a relative pronoun.

The accessibility hierarchy, and how it is obtained, is described ingreater detail in “Language Typology and Syntactic Description, ComplexConstructions”, Chapter 3, written by Edward L. Keenan, edited byTimothy Chopin, 1985; and Keenan, E. L. and B. Comrie, “N.P.Accessibility and Universal Grammar”, Linguistic Inquiry 8: 63-100(1977).

It is also worth noting that some languages make extensive use of whatlinguists broadly refer to as “cases”. The English language stillcontains vestiges of an earlier case system. For example, in thepronominal system, the English language distinguishes subject versusobject versus genitive with terms such as he, him and his. Whilelinguists have devoted a great deal of time and effort in attempting todistinguish case from thematic role from other kinds of marking, theterm case, as discussed herein, is used in the following two senses:

1. To describe morphological inflection, which typically involveschanging the endings of words. German, Russian and Latin are examples oflanguages which exhibit morphological case.

2. To describe the use of adpositions (prepositions and postpositions)or particles to indicate the grammatical role of a noun phrase. Japaneseand Indonesian are examples of languages which exhibit case informationof this type. A discussion of grammatical relations and surface case isset out in Shibatani, “Grammatical Relations and Surface Cases”,Language, Volume 53, Number 4 (1977) pp. 789-809. Also, a discussion ofgrammatical function and morphological case is set out in Maling, “OfNonminitive and Accusative: The Hierarchical Assignment of GrammaticalCase in Finnish”, this article is published in A. Holenberg and U.Nikanne, Case and Other Topics in Finnish Syntactic, Studies inGenerative Grammar, Foris (1992), pp. 51-76. In this patent, the term“relations” will be used to refer to both cases and grammaticalrelations.

SUMMARY OF THE INVENTION

One aspect of the present invention is related to the realization thatrelations are also generally indicative of the importance of a term orconstituent in determining what a sentence or document is about. Forexample, the higher the term is in the accessibility hierarchy, the moreimportant it is to the meaning of the sentence or document in which itexists. Similarly, certain case markings can indicate such importance.This can be extremely helpful in information retrieval type tasks.

A system is utilized for determining a relationship between first andsecond textual inputs. The system identifies relations of (i.e., caseinformation associated with, or grammatical relations of) constituentsin the first textual input. The relationship between the first andsecond textual inputs is then determined based on the relationsidentified. The grammatical relations can be determined based on theAccessibility Hierarchy. For example, constituents below a hierarchythreshold, determined based on the grammatical relations, can beeliminated from the first textual input, weighted in the first textualinput, or simply annotated. Similarly, constituents being marked bycertain case information can be eliminated from the first textual input,weighted, or simply annotated.

One embodiment of the invention includes a test methodology which isused in identifying the hierarchy and hierarchy threshold or therelevant case markings. The test methodology can be used across a widevariety of content sets, in order to customize the present invention foruse with the various content sets.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high level block diagram of an illustrative system in whichthe present invention can be used.

FIG. 2 is a block diagram of one embodiment of a computer in accordancewith one aspect of the present invention.

FIG. 3A is a functional block diagram illustrating operation of oneaspect of the present invention, in accordance with one illustrativeembodiment thereof.

FIG. 3B is a flow diagram illustrating operation of the system shown inFIG. 3A.

FIG. 4 is a flow diagram illustrating the operation of the system shownin FIG. 3A in accordance with another embodiment of the presentinvention.

FIG. 5A is a functional block diagram illustrating another system inwhich the present invention can be used.

FIGS. 5B-5F are flow and block diagrams illustrating the operation ofthe system shown in FIG. 5A.

FIG. 6A is a block diagram illustrating another system in which thepresent invention can be used.

FIGS. 6B and 6C are flow diagrams illustrating the operation of thesystem shown in FIG. 6A in accordance with one embodiment of the presentinvention.

FIG. 7 is a flow diagram illustrating a test methodology used inaccordance with one embodiment of the present invention.

DETAILED DESCRIPTION OF THE ILLUSTRATIVE EMBODIMENTS

Overview

The present invention utilizes grammatical relations of terms or phrases(constituents) in a first textual input in order to determine themeaning or “aboutness” of the first textual input. In accordance withone illustrative embodiment of the present invention, a hierarchy ofgrammatical relations is identified wherein grammatical relationsfalling below a predetermined hierarchy level or threshold are notparticularly helpful in determining the aboutness of the first textualinput. In another illustrative embodiment, case information isidentified which indicates that constituents corresponding to that caseinformation are not particularly helpful in determining the aboutness ofthe first textual input. For purposes of this discussion, the term“relations” is used to refer to both grammatical relations and cases.While the present invention is contemplated for use in a wide variety ofapplications, it is described herein primarily in the context ofinformation retrieval, for the purpose of illustration only. In thatcontext, the constituents which have been identified can be removed fromthe first textual input prior to an indexing operation to greatly reducethe size of the index. This can be accomplished without significantlyimpacting either precision or recall during the information retrievalprocess. The present invention also includes a system for obtaining alist of the types of constituents to be identified in identifying therelevant case information or in setting the hierarchy and hierarchythreshold, certain data structures and systems for generating those datastructures.

FIG. 1 depicts a very high-level block diagram of computer system 10 inaccordance with one illustrative embodiment of the present invention.Computer system 10 includes data store 12 coupled to computer 20. Itshould be noted that data store 12 can be any suitable type of storagewhich is either internal to, or external to, computer 20. Similarly,data store 12 can be coupled to computer 20 through any suitable means,such as through direct cabling, a dial-up modem connection (such as whendata store 12 corresponds to memory accessible over a global computernetwork such as the Internet), or any other suitable connection.Similarly, computer 20 is illustratively any suitable computer foraccessing data store 12, and is referred to hereinafter as a personalcomputer which is described in greater detail with respect to FIG. 2.

Computer 20 includes user interface devices (such as a keyboard andmonitor) so that a user can provide a user input to computer 20 in orderto request that computer 20 perform certain operations on data store 12.

The user input can be a request to perform an information retrievaloperation. In that case, the user input is typically a query andcomputer 20 executes the query against data store 12 to locate documentswhich are relevant to the query input by the user. Computer 20determines a relationship between the first textual input (which is thequery) and a second textual input (which is one or more of the documentsunder consideration or indexed records of those documents) andidentifies those documents which have a close relationship (such asthose documents which are similar in meaning) to the query and providesthe identification of those documents, or the documents themselves, asan output to the user.

The user input can also be an implicit type of information retrievalrequest. In that case, the user typically presents a document or othertextual input and requests computer 20 to execute a “FIND SAME” type ofoperation in which computer 20 locates documents on data store 12 whichare similar to the textual input provided by the user. Computer 20 thenprovides either an identification of the documents located in responseto the “FIND SAME” request, or the documents themselves.

Similarly, the user input can include a request to cluster documents, inwhich case computer 20 examines a plurality of documents on data store12 and clusters like documents in the same clusters. In doing so,computer 20 determines a relationship (such as similarity in meaning)between the various documents and places those documents which areclosely related to one another in the same cluster.

The user input can also be a request that computer 20 summarize one ormore documents contained in data store 12. In that instance, computer 20examines the documents under consideration and develops an output to theuser which is indicative of the meaning or “aboutness” of the documentbeing summarized.

In all of the above instances, it should be noted that the operationsdescribed can either be carried out across a network (either a localarea network or a wide area network) or they can be carried out within asingle computer (such as where the operating system provides a searchingtool which can be used to search the memory of the computer on which theoperating system is running).

FIG. 2 is a more detailed block diagram of computer 20 in accordancewith one illustrative embodiment of the present invention. FIG. 2 andthe related discussion are intended to provide a brief, generaldescription of a suitable computing environment in which the inventionmay be implemented. Although not required, the invention will bedescribed, at least in part, in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a personal computer. Generally, program modules includeroutine programs, objects, components, data structures, etc. thatperform particular tasks or implement particular abstract data types.Moreover, those skilled in the art will appreciate that the inventionmay be practiced with other computer system configurations, includinghand-held devices, multiprocessor systems, microprocessor-based orprogrammable consumer electronics, network PCs, minicomputers, mainframecomputers, and the like. The invention may also be practiced indistributed computing environments where tasks are performed by remoteprocessing devices that are linked through a communications network. Ina distributed computing environment, program modules may be located inboth local and remote memory storage devices.

In FIG. 2, an exemplary system for implementing the invention includes ageneral purpose computing device in the form of a conventional personalcomputer 20, including processing unit 21, a system memory 22, and asystem bus 23 that couples various system components including thesystem memory to the processing unit 21. The system bus 23 may be any ofseveral types of bus structures including a memory bus or memorycontroller, a peripheral bus, and a local bus using any of a variety ofbus architectures. The system memory includes read only memory (ROM) 24a random access memory (RAM) 25. A basic input/output 26 (BIOS),containing the basic routine that helps to transfer information betweenelements within the personal computer 20, such as during start-up, isstored in ROM 24. The personal computer 20 further includes a hard diskdrive 27 for reading from and writing to a hard disk (not shown), amagnetic disk drive 28 for reading from or writing to removable magneticdisk 29, and an optical disk drive 30 for reading from or writing to aremovable optical disk 31 such as a CD ROM or other optical media. Thehard disk drive 27, magnetic disk drive 28, and optical disk drive 30are connected to the system bus 23 by a hard disk drive interface 32,magnetic disk drive interface 33, and an optical drive interface 34,respectively. The drives and the associated computer-readable mediaprovide nonvolatile storage of computer readable instructions, datastructures, program modules and other data for the personal computer 20.

Although the exemplary environment described herein employs a hard disk,a removable magnetic disk 29 and a removable optical disk 31, it shouldbe appreciated by those skilled in the art that other types of computerreadable media which can store data that is accessible by a computer,such as magnetic cassettes, flash memory cards, digital video disks,Bernoulli cartridges, random access memories (RAMs), read only memory(ROM), and the like, may also be used in the exemplary operatingenvironment.

A number of program modules may be stored on the hard disk, magneticdisk 29, optical disk 31, ROM 24 or RAM 25, including an operatingsystem 35, one or more application programs 36, other program modules37, and program data 38. A user may enter commands and information intothe personal computer 20 through input devices such as a keyboard 40 andpointing device 42. Other input devices (not shown) may include amicrophone, joystick, game pad, satellite dish, scanner, or the like.These and other input devices are often connected to the processing unit21 through a serial port interface 45 that is coupled to the system bus23, but may be connected by other interfaces, such as a sound card, aparallel port, a game port or a universal serial bus (USB). A monitor 47or other type of display device is also connected to the system bus 23via an interface, such as a video adapter 48. In addition to the monitor47, personal computers may typically include other peripheral outputdevices such as a speaker and printers (not shown).

The personal computer 20 may operate in a networked environment usinglogic connections to one or more remote computers, such as a remotecomputer 49. The remote computer 49 may be another personal computer, aserver, a router, a network PC, a peer device or other network node, andtypically includes many or all of the elements described above relativeto the personal computer 20, although only a memory storage device 50has been illustrated in FIG. 1. The logic connections depicted in FIG. 1include a local are network (LAN) 51 and a wide area network (WAN) 52.Such networking environments are commonplace in offices, enterprise-widecomputer network intranets and the Internet.

When used in a LAN networking environment, the personal computer 20 isconnected to the local area network 51 through a network interface oradapter 53. When used in a WAN networking environment, the personalcomputer 20 typically includes a modem 54 or other means forestablishing communications over the wide area network 52, such as theInternet. The modem 54, which may be internal or external, is connectedto the system bus 23 via the serial port interface 46. In a networkenvironment, program modules depicted relative to the personal computer20, or portions thereof, may be stored in the remote memory storagedevices. It will be appreciated that the network connections shown areexemplary and other means of establishing a communications link betweenthe computers may be used.

Identifying Relations

FIG. 3A is a functional block diagram of a text processing system 100 inaccordance with one illustrative embodiment of the present invention.The components in system 100 can be implemented based on program modulesor instructions stored in any suitable memory associated with, oraccessible by, computer 20. Further, the components can be implementedin processor 21, a co-processor, or a dedicated processor.

Text processing system 100 includes constituent marking component 102and relationship determination component 104. Constituent markingcomponent 102 receives a first textual input and identifies constituenttypes and associated case information or grammatical relations of theconstituents in the first textual input, which are described in greaterdetail below. When case information is identified, the associatedconstituents are annotated so they can be compared against predeterminedcase information during later processing. When grammatical relations areidentified, the constituents are annotated so that they can be rankedduring later processing.

When grammatical relations are identified, the annotated constituentsand grammatical relations are provided, in one illustrative embodiment,as a syntactic parse tree or other syntactic analysis, to relationshipdetermination component 104, although other types of analysis whichidentify grammatical relations of constituents can be used as well. Whencase information is identified, the associated constituent can simply beprovided with an annotation which is indicative of the relation.

In an embodiment in which system 100 is being used to determine therelationship between two or more textual inputs, relationshipdetermination component 104 determines a relationship between the firsttextual input, as represented by the syntactic analysis or annotatedconstituents output by marking component 102 and a second textual inputprovided to component 104. Component 104 provides an indication of therelationship at its output.

In order to identify and annotate the constituents, marking component102 includes case or grammatical relation analyzer 106, and constituentidentifier and annotator 108. The first textual input is provided toanalyzer 106. In one illustrative embodiment, in which case informationis to be obtained, analyzer 106 includes a word breaker (e.g., forJapanese) and an analyzer which determines where case marking occurs. Inanother embodiment in which morphological case information is to beidentified (e.g., German), analyzer 106 includes a part-of-speechanalyzer and a morphological analyzer which analyzes words formorphological inflections. The constituents with associated caseinformation are provided as an output to constituent identifier andannotator 108.

In another illustrative embodiment, analyzer 106 simply includes asentence breaker which examines the first textual input provided theretoand breaks the document along sentence boundaries. The sentences areprovided to a syntactic and semantic analyzer (also included ingrammatical relation analyzer 106) which identifies parts of speech(such as noun and verb), constituents (such as noun phrases and verbphrases), and grammatical relations (such as subjects and objects), andprovides an output based on a syntactic analysis which corresponds tothe sentence received by the syntactic and semantic analyzer. It shouldbe noted that any other suitable mechanism which identifies grammaticalrelations among constituents can be used as well.

The syntactic analysis or other output (such as constituents with caseinformation) is provided to constituent identifier and annotator 108. Itshould also be noted that grammatical analyzer 106 can be a morecomplicated system which includes other morphological analysis, or othernatural language processing techniques as well.

For example, in accordance with one more specific illustrativeembodiment of the present invention, the syntactic and semantic analyzerin grammatical relation analyzer 106 receives a single line of inputtext at any given time, whether it be a sentence in a document or a textfragment and constructs a syntactic analysis which serves as input tothe remainder of the grammatical relation analyzer 106. In oneillustrative example, where documents are being analyzed, in order tofacilitate natural language processing which operates on a singlesentence at a time, the text for each document is broken into a textfile using a conventional sentence breaker in which each sentenceoccupies a separate line in the file. For each input text line, the lineis first parsed into its constituent words. Thereafter, using apredefined record in a stored lexicon, for each word, the correspondingrecords for these constituent words (through predefined grammaticalrules) are themselves combined into larger structures or analyses whichare then, in turn, combined (again through predefined grammatical rules)to form even larger structures, such as a syntactic parse tree. Whethera particular rule will be applicable to a particular set of constituentsis governed, in part, by the presence or absence of certaincorresponding attributes and their values in the word records.

An illustrative lexicon includes approximately 165,000 head wordentries. This lexicon includes various classes of words (such as, e.g.,prepositions, conjunctions, verbs, nouns, operators and qualifiers) thatdefine syntactic and semantic properties inherent in the words in aninput string so that a parse tree can be constructed therefore. Clearly,a syntactic parse tree can be precomputed while a corresponding documentis being indexed and stored within a record for that document forsubsequent access and use, rather than being computed later once thatdocument has been retrieved in response to a query.

In one particular illustrative embodiment of grammatical analyzer 106,an input string, such as the sentence “The octopus has three hearts.” isfirst morphologically analyzed, using the predefined record in thelexicon for each of its constituent words, to generate a so-called“stem” (or “base”) form. Stem forms are used in order to normalizediffering word forms (e.g., verb tense and singular-plural nounvariations) to a common morphological form for use by a parser. Once thestem forms are produced, the input string is syntactically analyzed bythe parser, using the grammatical rules and attributes in the records ofthe constituent words, to yield the corresponding syntactic parse tree.This tree depicts the structure of the input string, specifically eachword or phrase (e.g. noun phrase “The octopus”) in the input string, acategory of its corresponding grammatical function (e.g., NP for nounphrase) and link(s) to each syntactically related word or phrasetherein. For the illustrative sentence, its associated syntactic parsetree would be: TABLE 1 SYNTACTIC PARSE TREE for “The octopus has threehearts.”

A start node located in the upper-left hand corner of the tree definesthe type of input string being parsed. Sentence types include “DECL” (ashere) for a declarative sentence, “IMPR” for an imperative sentence and“QUES” for a question. Displayed vertically to the right and below thestart node is a first level analysis. This analysis has a head nodeindicated by an asterisk (or other suitable marker), typically a mainverb (here the word “has”), a premodifier (here the noun phrase “Theoctopus”), followed by a postmodifier (the noun phrase “three hearts”).Each leaf of the tree contains a lexical term or a punctuation mark.Here (as labels), “NP” designates a noun phrase, and “CHAR” denotes apunctuation mark.

About this and subsequent levels of processing, the reader is referredto co-pending U.S. patent applications entitled “METHOD AND SYSTEM FORCOMPUTING SEMANTIC LOGICAL FORMS FROM SYNTACTIC TREES”, filed Jun. 28,1996 and assigned Ser. No. 08/674,610, and “INFORMATION RETRIEVALUTILIZING SEMANTIC REPRESENTATION OF TEXT”, filed Mar. 7, 1997 andassigned Ser. No. 08/886,814, both of which have been assigned to thepresent assignee and incorporated by reference herein.

The output of case or grammatical relation analyzer 106 (illustrativelya syntactic analysis corresponding to the sentence under analysis) isprovided to constituent identifier and annotator 108. In an illustrativeembodiment in which constituents and associated case information areprovided by analyzer 106, constituent identifier and annotator 108includes a predetermined list of case information which identifies caseswhich tend to be less indicative of the aboutness of the first textualinput. In such an embodiment, component 108 includes a softwarecomponent which receives the output from analyzer 106 and looks forconstituents having associated case information which is contained inthe predetermined list. Component 108 annotates those constituents whichcontain the predetermined case information (and thus annotates thoseconstituents which are relatively less related to the aboutness of thedocument).

In an illustrative embodiment, constituent identifier and annotator 108includes a predetermined hierarchy of grammatical relations to beidentified and a predetermined hierarchy threshold. In one illustrativeembodiment, the predetermined hierarchy is the Accessibility Hierarchyset out by Keenan & Comrie and discussed above in the Backgroundsection. The threshold is simply defined by a predetermined level orposition in the hierarchy. Therefore, all grammatical relations residingbelow the threshold position in the hierarchy are contained in apredetermined list of grammatical relations. Component 108 annotates theconstituents whose grammatical relation falls below the threshold. Thus,constituent identifier and annotator 108 includes a software componentwhich traverses the syntactic analysis provided by grammatical relationanalyzer 106 looking for constituents whose grammatical relations arecontained in the list of predetermined grammatical relations (and thuscorresponding to constituents whose grammatical relations fall below thethreshold). Upon identifying a grammatical relation in a sentence passedto the identifier and annotator 108 as one in the predetermined list,component 108 annotates the associated constituent to indicate that itis one whose grammatical relation is contained in the list ofpredetermined grammatical relations.

In accordance with one aspect of the present invention, thepredetermined grammatical relations or case information are those whichhave been empirically determined to be of little or no use indetermining what the overall document or sentence is about. For example,in many instances, the subject of a main clause in a sentence tends tocorrespond to a reader's intuitive reading of what the sentence ordocument is about, while an indirect object in a subordinate clause doesnot. Further, for Japanese, documents containing matched keywords markedwith a topic marker “wa” are very often highly relevant, while documentscontaining keywords with a subordinate marker “no” (which can mark e.g.Partitive, Attributive, Possessive, Appositive) are very oftenirrelevant. Similarly, in Japanese, several different nominalcoordination markers (“ya”, “nado”, and comma) are used only in anenumeration context (such as “A, B, C, etc”). Documents containingkeywords marked with these enumeration markers are very oftenirrelevant. Such constituent types and case information can beempirically identified according to a suitable test methodology, one ofwhich is described below in greater detail with respect to FIG. 7.

Further, it should be noted that complete syntactic analysis need not beimplemented for the crude identification of grammatical relations orcases. Instead, parts of speech and their location in a sentence can beidentified more simply. For example, it has been observed that nounslocated early in a sentence are often subjects of the sentence.Therefore, if the initial threshold in the Accessibility Hierarchy isset just below subjects, all terms in a sentence after the first nouncan be annotated. Other techniques can be used as well. Algorithms whichperform part of speech tagging are widely used and well known.

In any event, once the relevant constituents have been annotated, theannotated structure (such as an annotated syntactic analysis) is passedto relationship determination component 104. In the embodimentillustrated in FIG. 3A, component 104 is configured to determine apredetermined relationship between the first textual input provided toanalyzer 106 and a second textual input provided to component 104. Inone illustrative embodiment, the relationship to be determined is thesimilarity in meaning between the first and second textual inputs.

In performing that task, component 104 takes into account the annotatedconstituents (such as in the annotated syntactic analysis) provided byconstituent marking component 102. For example, component 104 can simplyeliminate the annotated constituents and determine a degree ofsimilarity between the first textual input (with the annotatedconstituents deleted) and the second textual input. This reduces theamount of comparison and processing required, since the overall text inthe first textual input is reduced. Weighting techniques can also beused and are described in greater detail later in the application.

FIG. 3B is a flow diagram illustrating in greater detail the operationof system 100 illustrated in FIG. 3A. First, case or grammaticalanalyzer 106 receives the first textual input. This is indicated byblock 110. In one illustrative embodiment, the first textual input is adocument under analysis. Analyzer 106 breaks the document at sentenceboundaries. In addition, in one illustrative embodiment, analyzer 106may contain morphological, syntactic and semantic analysis components,or contain no such components, but simply a component which identifiescase information or grammatical relations in each sentence. This isindicated by block 112.

Having generated the syntactic analysis (such as that illustrated inTable 1) or relation identifications, analyzer 106 passes that output toconstituent identifier and annotator 108. Component 108 comparesrelations identified against the predetermined list of relations ortraverses the syntactic analysis illustrated by Table 1 until itencounters a constituent whose relation is on the predetermined list ofgrammatical relations (and thus below the predetermined hierarchythreshold). Assuming that, for the example illustrated in Table 1,objects are on the list of predetermined grammatical relations,component 108 traverses the syntactic analysis until it encounters thenode which has been identified by grammatical analyzer 106 as an object.Since that grammatical relation appears in the predetermined list ofgrammatical relations, component 108 annotates that node (and optionallyall terms descending from that node) as being contained in the list ofpredetermined grammatical relations and thus falling below the hierarchythreshold. This annotation is illustrated in Table 1 as “/object”. Theidentification of predetermined grammatical relations and annotation ofthose grammatical relations is indicated by blocks 114 and 116 in FIG.3B. Again, the term “relations” as used herein refers to bothgrammatical relations and cases.

The annotated syntactic analysis (or output containing relationannotations) is then passed to component 104 in FIG. 3A. In anembodiment, for example, where system 100 is determining the similaritybetween the first and second textual inputs, component 104 can perform anumber of operations on the annotated output from component 102. Forexample, since it has been empirically determined that the annotatedterms do not closely correspond to what the first textual input isabout, all annotated terms can simply be pruned from the annotatedoutput. This is indicated by block 118 in FIG. 3B.

Once the annotated terms have been pruned, and the second textual inputhas been received, component 104 simply determines the similaritybetween the terms remaining in the pruned text and the terms in thesecond textual input. By way of example, where the first textual inputis a document and the second textual input is a query, the content wordsin the second textual input (the query) are compared against the wordsremaining in the pruned syntactic analysis corresponding to thesentences in the document in order to determine whether any matchesexist. A score is assigned to the document which is based on the numberof matches which occur and which is thus indicative of the similaritybetween the first textual input (the document) and the second textualinput (the query). This score is provided at the output of component 104for use in further processing. Returning the pruned syntactic analysis,receipt of the second textual input, and determining the relationshipbetween the first and second textual inputs are indicated by blocks 120,122 and 124 in FIG. 3B.

As discussed above, component 104 can perform any number of operationson the annotated output provided by marking component 102, other thansimply pruning the annotated terms. FIG. 4 is a flow diagramillustrating an alternative operation in greater detail. A number ofblocks are similar to those shown in FIG. 3B, and are similarlynumbered. Therefore, system 100 receives the first textual input andperforms analysis to identify case information or grammatical relationscorresponding to constituents. thereof (such as by generating asyntactic analysis), and also identifies whether the constituents haveassociated relations which correspond to the predetermined relations(such as by locating the relations in the predetermined list orhierarchy of relations). This is indicated by blocks 110, 112 and 114.

However, rather than simply annotating the identified constituents in abinary fashion (as described with respect to FIG. 3B), in theillustrative embodiment shown in FIG. 4, component 108 assigns aweighting value to each of the constituents based upon how closely eachconstituent is related to the “aboutness” of the first textual input.For example, it may be empirically learned that the subject of a matrixclause is more closely related to the aboutness of the document than anobject in a subordinate clause. Alternatively, it can simply be assumedthat constituents having associated grammatical relations residinghigher in the above-discussed Accessibility Hierarchy are more closelyrelated to aboutness than constituents which have associated grammaticalrelations located lower in the hierarchy. That being the case, terms inthe first textual input are preferentially matched against terms locatedhigher in the hierarchy. One way to accomplish this is to assign varyingweights to constituents, based on the location of their correspondingrelations in the hierarchy. Once the relations of constituents have beenidentified during the analysis of the textual input, objects insubordinate clauses may be assigned a very low value (e.g., 0.1 on ascale of 0 to 1) while subjects of matrix clauses may be weighted with avery high value (such as 0.9). The particular weighting value assignedcan be empirically, or intuitively, predetermined. Similar weighting canbe done based on case information identified. In other words,constituents having associated relations located high on a predeterminedranked list of relations will be given a higher weight than thoselocated lower on the list (assuming that the relation located higher onthe list is more closely related to aboutness). This is indicated byblock 116 in FIG. 4.

The annotated text (e.g., syntactic analysis) is then returned tocomponent 104, as indicated by block 126. Next, as described withrespect to FIG. 3B, the second textual input is received as indicated byblock 122, and the relationship between the first and second textualinputs is determined by referring to the annotated (weighted) text, asindicated by block 128.

Summarization and Clustering

The techniques in accordance with the present invention can be used in awide variety of applications. For example, where the application is tocreate a summary of a document, the present invention can be used toidentify, and eliminate from the summary, clauses or terms which are notrelated to the aboutness of the document. In that event, system 100 doesnot include relationship determination component 104 which is used indetermining the relationship between first and second textual inputs.Rather, a summary generation component is substituted therefore which isconfigured to generate a summary based on the annotated or pruned text(e.g., syntactic analysis) returned by component 102.

Similarly, where the application is a document clustering application,system 100 is configured to receive all of the documents to beclustered, and to create annotated texts (e.g., syntactic analyses)based on sentences in each of the documents. Then, relationshipdetermination component 104 simply takes, as its inputs, annotated textsfrom two or more documents to determine a relationship between the twodocuments. Relationship scores are computed for all possible pairs ofdocuments, and those with the highest relationship scores are clusteredtogether.

Information Retrieval Prior to Indexing

While all of the above applications can be related to the task ofinformation retrieval, the present invention can also illustratively bedirectly implemented in an application which performs informationretrieval within a computer, across a local area network, or across awide area network, such as a global computer network. FIGS. 5A-5Fillustrate use of the present invention in information retrievaloperations in the context of a global computer network, whereintechniques according to the present invention are performed prior toindexing.

FIG. 5A is a functional block diagram setting constituent markingcomponent 102 (described with respect to FIG. 3A) in the context ofInternet information retrieval system 200. System 200 includes webcrawler component 202, text filter component 204, constituent markingcomponent 102, indexer 206 and index 208. System 200 also illustrativelyincludes search engine 210. FIG. 5B is a flow diagram which illustratesthe operation of system 200.

Web crawler component 202 is a conventional component, illustrativelyintegrated in the operating system of a computer, which periodicallyaccesses predetermined sites on the global computer network andascertains whether any content has been added to the site which has beenaccessed. If so, web crawler component 202 provides the content to textfilter 204. Text filter 204 is also a conventional component whichremoves items which are undesirable from an indexing standpoint. Forexample, in one illustrative embodiment, text filter component 204removes all HTML tags, and other such items, from the informationprovided by web crawler 202. Receiving the web content information isindicated by block 212 in FIG. 5B, and filtering the text is indicatedby block 214 in FIG. 5B. The filtered information is then provided toconstituent marking component 102. Constituent marking component 102operates in a similar fashion to that described with respect to FIG. 3A.In other words, constituent marking component 102 illustrativelyincludes a sentence breaker which breaks the text provided at its inputat sentence boundaries. Component 102 also preferably includes case orgrammatical analysis functionality which parses the sentences andprovides an output which identifies relations (i.e., case information orgrammatical relations) and compares them against the predetermined,ranked list or places them in the predetermined hierarchy. Component 102also illustratively includes constituent identifier and annotator 108which receives the output provided by the case or grammatical relationanalyzer to annotate constituents based on their case information orposition in the predetermined hierarchy.

Constituents having associated predetermined case information orgrammatical relations residing below a predetermined hierarchy thresholdcan be pruned or some or all of the constituents can be annotated with aweighting value as discussed above. Identifying and annotating theconstituents is indicated by block 216 in FIG. 5B.

The terms in the annotated or pruned text are then provided to indexercomponent 206. Indexer 206 can take any number of suitable forms, suchas being incorporated in search engine 210 described below and asdescribed in greater detail with respect to FIGS. 5C-5F below. However,suffice it to say that indexer 206 generates an index corresponding tothe documents retrieved by web crawler component 200 and provides theindex 208 at its output. Index 208 can be a simple statistical,keyword-type index, or a more sophisticated type of index, dependingupon indexer 206. Generation of the index is indicated by block 218 inFIG. 5B.

Search engine 210 is configured to receive a user input query and toexecute that query against index 208. In one illustrative embodiment,search engine 210 also performs the indexing operation described withrespect to indexer 206. In such an embodiment, search engine 210 is anInternet search engine, such as Alta Vista (which is a registeredtrademark of Digital Equipment Corporation of Maynard, Mass.), and isconnected to a mass data store containing index 208. The data set ofdocument records is illustratively indexed by indexer 206 which can becontained in the search engine. Each such record in index 208 typicallycontains a web address (commonly referred to as a uniform resourcelocator “URL”) at which a corresponding document can be accessed by aweb browser, predefined content words which appear in that documentalong with, in certain engines, a relative address of such word relativeto other content words in that document; a short summary (often just afew lines) of the document or a first few lines of the document; and,possibly, the description of the document as provided in its hypertextmarkup language (HTML) description field.

When a user enters the query through, for example, a web browser, thebrowser passes the query through an Internet connection to a serverwhich contains the search engine 210. The search engine 210 processesthe query against document records stored within index 208 to yield aset of retrieved records (for documents) that the engine 210 determinesto be relevant to the query.

In one embodiment, search engine 210 compares the content words in thequery against content words in document records indexed in index 208.Based on the number of matches, the documents are assigned a score andranked. The ranked documents are output to the user by search engine210. Receiving and executing the query is indicated by blocks 220 and222 in FIG. 5B, and returning the documents is indicated by block 224.

Of course, it should be noted that search engine 210 can be a morecomplex type of search engine. In that case, search engine 210 mayperform natural language processing on the query, or on the returneddocuments, or on both, prior to providing any documents to the user.Other types of search engines are contemplated as well.

FIG. 5C is a more detailed block diagram of indexer 206 in accordancewith one illustrative embodiment of the present invention. In FIG. 5C,indexer 206 includes constituent pruner component 226 and indexgenerator component 228. In the embodiment illustrated in FIG. 5C, theindex 208 to be generated is simply a statistical keyword type index,and index generator 228 is a conventional index generator for generatingsuch an index based on a textual input. In that case, constituent pruner226 operates to prune, or strip out, all annotated constituents or termswhich are provided by constituent marking component 102 (and whichcontain the predetermined case information or which fall below thegrammatical relation hierarchy threshold), and to provide the remainingterms to index generator 228. Therefore, all aspects of system 200(other than constituent marking and pruning, but including indexgeneration) can be accomplished with conventional components. Thetextual inputs are simply provided to marking component 102, and pruner226, after they have been filtered by component 204, and prior toindexing by index generator 228. Components 102 and 226 act to identifyand eliminate from indexing, terms and clauses which have little or norelationship to the aboutness of the document being indexed. Thissignificantly reduces the size of index 208, without compromisingperformance across the precision and recall dimensions.

FIG. 5D is a flow diagram which illustrates the operation of indexer 206shown in FIG. 5C. Pruning the annotated constituents is indicated byblock 230 in FIG. 5D, and indexing the remaining terms usingconventional indexing techniques is indicated by block 232 in FIG. 5D.

FIG. 5E is a functional block diagram illustrating another illustrativeembodiment of indexer 206, and FIG. 5F is a flow diagram illustratingthe operation of indexer 206 shown in FIG. 5E. Indexer 206 in FIG. 5Eincludes constituent weighting component 234 and index generator 236.Constituent weighting component 234 receives the annotated output fromcomponent 102 and weights the terms in the annotated output, or weightscertain of the terms. The weighting can be accomplished using a binarytype scale or a continuous scale.

For example, term weighting component 234 can be configured to simplyadd a predesignated marking to all words in an annotated syntacticanalysis. In other words, all words in the syntactic analysis which havegrammatical relations located above the hierarchy threshold can beprepended with a “g_” indicating that those words represent a goodmatch. Similarly, all annotated words contained within the syntacticanalysis (and thus having grammatical relations located below thehierarchy threshold) can be prepended with a “b” indicating that theycorrespond to a bad match, or a match which does not necessarilyindicate that the two textual inputs are similar. Similar markings canbe made based on the predetermined, ranked list of case information.

By way of example, assume the grammatical relation hierarchy thresholdis set such that all terms in the hierarchy below subjects are below thethreshold. Then, if “dog” is a word contained in the syntactic analysiswhich is a subject, term weighting component 234 receives the term “dog”and generates at its output the term “g_dog”, which is indexed.Similarly, if the term “bone” is located within the syntactic analysisand is a direct object, then weighting component 234 receives the term“bone” at its input and generates at its output the term “b_bone”. Allof these terms are provided to index generator 236 which, in thatembodiment, simply generates a statistical keyword type index using thewords with the prepended terms.

Then, when the user provides the query to search engine 210, searchengine 210 is configured to prepend each content word in the query withthe term “g_”. Search engine 210 then identifies matches between onlyterms in the index and query which represent a good match. Thus,preferential matching is accomplished against only terms which aresubjects or topic words in the document.

Again, it should be noted that index generator 236 can, in thatembodiment, be implemented as a conventional index generator whichsimply generates a keyword-type index 208 without any specialmodification. In the above-described embodiment, index 208 does notachieve the space reduction advantages which are achievable using thepresent invention, but it does achieve increased precision advantages inaccordance with the present invention.

In an alternative embodiment, term weighting component 234 weights theterms in the annotated output received from component 102 with acontinuous, or semi-continuous weighting scheme. In that event, termweighting component 234 is configured to place a weight by eachannotated constituent indicating the expected relevance which theconstituent has to the aboutness of the document, based on empiricalanalysis of case, or based on its position in the predeterminedgrammatical relation hierarchy. For example, if empirical results showthat a certain constituent having associated grammatical relationlocated high in the predetermined hierarchy is highly related to theaboutness of a document, a weight can be appended to that constituentwhich indicates, for example, a 90% confidence level that the term isrelevant. This weight is added to the terms in the annotated syntacticanalysis by term weighting component 234.

Index generator 236, in that embodiment, is configured to create index208 which not only includes the terms provided to index generator 236,but which also includes the weight value added by term weightingcomponent 234. Thus, search engine 210 is configured to not only lookfor matches between content words in the query and those contained inindex 208, but to compute the score for similarity between the documentand the query based on the number of matches found (and any otherdesirable criteria) and also based on the weight assigned to thematching terms.

In another alternative embodiment, an existing indexing technique isused, with the addition of one item. A number of existing indexingtechniques index a document based on the word, the position of the wordwithin the document, and the capitalization of the word (i.e., whetherthe word is capitalized or not capitalized). This type of indexingscheme can be used in accordance with the present invention includingterm weighting component 234 by simply adding one additional entry intothe index. That entry corresponds to the weight value assigned by termweighting component 234. The weight can be empirically determined, orcan be determined using a conventional technique such as termfrequency-inverse document frequency (tf*idf). In any event, termweighting component 234 provides the weight value to index generator 236which builds the index based on the term, the position, thecapitalization and the assigned weight value, and any other conventionalentries used in the indexing scheme. Weighting the terms or constituentsbased on the annotation is indicated by block 238 in FIG. 5F, andgeneration of the index including the weighting information is indicatedby block 240.

Information Retrieval Operations, Performed on Documents Returned inResponse to a Query

The present invention can also be used in post processing operations(i.e., in operations which occur upon the document set returned by aconventional search engine in response to a query). FIG. 6A is afunctional block diagram illustrating constituent marking component 102in the context of a post processing information retrieval system 300.System 300 includes a conventional information retrieval engine 302which can be implemented as a simple statistical information retrievalengine, or as one which includes more complex natural languageprocessing techniques. Data store 304 represents the data store beingsearched based upon a user query input to information retrieval engine302. Data store 304 can, for example, be within a computer, on a localarea network, or on a wide area network (such as a global computernetwork). Similarly, data store 304 can simply be an index of documentsto be searched. System 300 also includes constituent pruner component306 or, optionally, constituent weighting component 308. System 300further includes matching component 310 and re-ranking component 312.

FIG. 6B is a flow diagram which illustrates the operation of system 300shown in FIG. 6A.

In operation, information retrieval engine 302 first receives a queryinput by the user. This is indicated by block 314 in FIG. 6B. Next,information retrieval engine 302 executes the query against the datastore 304 in a conventional manner. This is indicated by block 316 inFIG. 6B. Similarly, in a conventional manner, information retrievalengine 302 receives and returns the documents which are deemed by engine302 to be relevant to the user's query. This is indicated by block 318.

In accordance with one embodiment of the present invention, the returneddocuments are then provided to constituent marking component 102 whichis similar to that described with respect to FIG. 3A. The annotatedconstituents are then provided, in one illustrative embodiment, toconstituent pruner 306. Pruner 306 prunes the annotated constituentsfrom the annotated output provided by marking component 102. The prunedoutput is then provided to matching component 310. Identifying andannotating the constituents based on case or the location of associatedgrammatical relation in the predetermined hierarchy is indicated byblock 320 in FIG. 6B, and pruning those constituents is indicated byblock 322.

Matching component 310 selects a first of the returned documents, withthe annotated constituents removed therefrom. This is indicated by block324. Matching component 310 then determines whether any of the contentterms in the query provided by the user are still in the selecteddocument, after the annotated constituents have been removed. If not,that indicates that the only matches against the selected documentoccurred with respect to constituents which contained the predeterminedcase information or which had grammatical relations located below thepredetermined hierarchy threshold (i.e., those which have little to dowith describing what the document is about). This is indicated by block326 in FIG. 6B.

Therefore, the document is eliminated from the list of returneddocuments, as indicated by block 328, and matching component 310determines whether any documents remain in the returned document set.This is indicated by block 330.

If, at block 326, some of the content terms in the query are still foundin the selected document, after the annotated constituents have beenremoved therefrom, that indicates that matches have been made againstthe document with respect to terms that are believed to be relevant tothe aboutness of the document. Therefore, the selected document isretained as indicated by block 332. Again, matching component 310determines, at block 330, whether any of the returned documents have yetto be analyzed. If so, processing returns to block 324.

After all of the returned documents have been analyzed, the remainingdocuments (those which have not been eliminated) are passed tore-ranking component 312. Re-ranking component 312 first determineswhether the number of remaining documents is sufficient to present tothe user. This is indicated by block 334. In other words, it is possiblethat the content terms of the query matched relevant terms in only avery small number of the returned documents. If that is true, re-rankingcomponent 312 may determine that the number of documents presented tothe user is undesirably low, or is insufficient. This threshold caneither be set by the user, or can be automatically determined based on athreshold value preprogrammed into component 312.

In any event, if the number of documents remaining is sufficient, asdetermined at block 334, then re-ranking component 312 computes a newscore for the retained documents. The new score can be computed in anysuitable fashion. For example, the new score may be based on the numberof matches between content words in the query and terms remaining ineach of the documents, after the annotated constituents have beenremoved. Computation of the new score is indicated by block 336. Basedupon the new score, re-ranking component 312 re-ranks the retaineddocuments in an order determined by the new score. This is indicated byblock 338. Finally, component 312 returns the re-ranked documents to theuser in re-ranked order. This is indicated by block 340.

If, at block 334, it is determined that an insufficient number ofdocuments remains to present to the user, then processing continues atblock 342. Re-ranking component 312 retrieves all documents returnedfrom information retrieval engine 302 in response to the query input bythe user. Instead of removing the annotated constituents as indicated byblock 322, re-ranking component 312 feeds the documents back to optionalconstituent weighting component 308. Weighting component 308 weights theannotated constituents which had previously been removed (and may weightall other constituents as well) based on their case or positions oftheir grammatical relations in the predetermined hierarchy. The weightedoutputs are then fed back to matching component 310 which again matchesthe content words in the query against the terms in the documents,including annotated terms which have been weighted. This is indicated byblock 344. A new score is then computed for each of the returneddocuments based on the weights, and the documents are again re-ranked.This is indicated by block 346. The documents are then provided to theuser as indicated by block 340.

It should also be noted that, at block 344, instead of weighting theannotated constituents, constituent weighting component 308 can simplyremove the annotations which essentially weight all the terms the same.Then, matching component 310 simply matches against the same terms whichinformation retrieval engine 302 matched against. Further, rather thanremoving the annotations from all documents, the annotations can simplybe removed at block 344 from those documents which were eliminated fromthe search at block 328. Therefore, the annotations are simply removedfrom those documents which information retrieval engine 302 had alreadydetermined were relevant enough to return to the user. Those documentsare then simply ranked lower than the documents which remained even withthe annotated constituents removed.

It should be noted that the discussions herein which relate toinformation retrieval proceed using the term “content word.” This termis used for the sake of illustration only. It is contemplated that theparticular linguistic unit under analysis can be any other unit, such asmorphemes, words, phrases, n-grams of characters, etc., all of which arereferred to herein as “search terms.”

Test Methodology

One embodiment of the present invention also includes a system by whichthe pertinent case information (or ranked list of case information andcorresponding threshold) or grammatical relation hierarchy and hierarchythreshold can be predetermined based on any given data set with whichthe present invention is utilized. FIG. 7 is a flow diagram whichillustrates operation of computer 20 in implementing such a system.While the technique described below can be used to identify caseinformation and both the grammatical relation hierarchy to be used andthe hierarchy threshold, the following discussion proceeds primarilywith respect to setting the hierarchy threshold with respect to apredefined hierarchy. It should also be noted that a content set can beany suitable collection of documents or any other text corpus which canbe searched using information retrieval-type techniques.

First, a grammatical relation hierarchy threshold is selected. Thehierarchy against which the threshold is set can be any suitablehierarchy such as the Accessibility Hierarchy described above or anotherempirically determined or intuitively determined hierarchy or onedetermined using the same technique as that described herein for settingthe threshold. The initial threshold can be set essentially any positionalong the hierarchy.

It is believed that the particular hierarchy and hierarchy thresholdwhich may prove to be helpful in performing techniques in accordancewith the present invention may change with different content sets.Assuming a hierarchy has been selected, in order to identify thehierarchy threshold to be used in accordance with the present invention,as the predetermined hierarchy threshold, a first hierarchy threshold tobe tested is selected. This is indicated by block 350 in FIG. 7.

Next, an index is built from a predetermined text corpus with all of theconstituents having associated grammatical relations falling below theselected hierarchy threshold removed therefrom. This is indicated byblock 352. In accordance with one illustrative embodiment of the presentinvention, the text corpus is approximately 20,000-30,000 documentspertaining to the desired content set.

Next, information retrieval is performed on the index. This is indicatedby block 354. In one illustrative embodiment, information retrieval isperformed using approximately 20,000-30,000 predetermined queries.

Next, constituents in the content set which have an associatedgrammatical relation falling below the selected threshold are identifiedand a determination is made as to how useful the remaining constituents(other than those falling below the threshold) are in determining themeaning of the content set. Of course, this can be done in any number ofways such as observing knowledge about the contact set gained by readersof the constituents which reside above the threshold.

Another way is to use information retrieval. This discussion proceedswith respect to an illustrative embodiment which uses informationretrieval. In that embodiment, computer 20 determines how much of anindex space saving has been realized by removing the constituents havinggrammatical relations falling below the selected threshold from thedocument corpus. This is indicated by block 356. Computer 20 thendetermines the performance of the information retrieval operationsperformed against the index. This is indicated by block 358. In order todetermine whether performance is adequate, any suitable measure whichincludes precision and recall components can be used. One suitablemeasure is known as “IF-measure” and was defined in a paper written byC. J. Van Rijsbergen, 1979, entitled “INFORMATION RETRIEVAL.” TheF-measure is defined as follows:F=((B ²+1.0)*P*R)/(B ² *P+R)

where P is precision, R is recall and B is the relative importance givento recall over precision. Usually, B=1.

Next, computer 20 determines whether the performance is adequate giventhe reduction in index size. This will differ, of course, withapplication. In applications in which index size is of great importance,a greater reduction in information retrieval performance may betolerated. For purposes of the present discussion, a statisticallysignificant change in F-measure can be defined by a change of fivepercent or more.

In any event, it is decided whether the performance is adequate givenreduction in index size at block 360. If so, the selected threshold islogged and grammatical relations falling below the threshold are addedto the list of predetermined grammatical relations associated with thecontent set under analysis. This is indicated by block 362.

However, if, at block 360, it is determined that performance is notadequate given the reduction in index size, failure analysis isperformed to determine why documents were not returned that should havebeen, or why documents were returned that should not have been. This isindicated by block 364. Illustratively, this is determined by a group oflinguists who analyze the documents returned, or documents which shouldhave been returned, but were not, to determine what the documents havein common, and hence to determine why the documents were not returned,or were incorrectly returned. In performing this analysis, the linguistsmay add some further refinement to the hierarchy threshold (or hierarchystructure) such that constituents are only removed under certaincircumstances, or in certain contexts, even if they have associatedgrammatical relations which fall below the threshold. Similarly, thelinguists may simply determine that the selected threshold (or hierarchystructure) is not desirable, given the content set. In any case,computer 20 then determines whether additional thresholds (or hierarchystructures) are to be tested. If so, processing returns to block 350 anda new threshold is selected. If not, all thresholds which are to betested have been tested, and the predetermined threshold has beenselected. This is indicated by block 366.

The hierarchy structure can also be determined according to this generalprocess. Constituents having a selected grammatical relation are removedfrom the corpus and a quantitative indication of usefulness of theremaining constituents is obtained (such as by performing informationretrieval on an index formed based on the pruned corpus). Thegrammatical relation which has the greatest usefulness (e.g., results inthe greatest reduction in information retrieval performance) is arrangedhighest in the hierarchy. Remaining grammatical relations are arrangedin descending order, based on reduction in usefulness (e.g., a reductionin information retrieval performance).

Further, this general process can be used to select case information tobe used in the predetermined, ranked list. For example, in theillustrative information retrieval embodiment, a particular case isselected, terms having the selected case are annotated and removed, andan index is built with the remaining terms. Information retrieval isperformed on the index and performance is analyzed. Based on a reductionin performance, cases are placed in a ranked list. The threshold withinthe ranked list can also be set based on reduction in informationretrieval performance. Further analysis can be performed whereperformance is inadequate. The process can be repeated for any desiredcase information.

Thus, it can be seen that the present invention identifies relations(case information or a grammatical relation hierarchy and a hierarchythreshold). Constituents with the associated case information or havinggrammatical relations located below the hierarchy threshold areidentified so they can be annotated, weighted, or simply removed whendetermining what a document is about. This can be utilized when adocument is indexed during information retrieval, when a document isbeing summarized, after a document has been retrieved using conventionalinformation retrieval techniques, when documents are to be clustered, orwith an implicit query in which a document is provided and a computer isrequested to find similar documents. The present invention can beutilized in other contexts as well. However, it has been found to beparticularly beneficial in information retrieval indexing since thepresent invention can greatly reduce the index size withoutsignificantly sacrificing performance.

Although the present invention has been described with reference topreferred embodiments, workers skilled in the art will recognize thatchanges may be made in form and detail without departing from the spiritand scope of the invention.

1. A computer readable medium storing instructions which, when executedby a computer, cause the computer to perform a method of determining arelationship between first and second textual inputs, the methodcomprising: obtaining a hierarchy of relations; obtaining a hierarchythreshold based on a usefulness of the relations in the hierarchy indetermining the relationship between the first and second textualinputs; identifying constituents in the first textual input that havethe relations in the hierarchy; determining a usefulness of theidentified constituents by locating the identified constituents in thehierarchy; and determining the relationship between the first and secondtextual inputs based on identified constituents having associatedrelations above the hierarchy threshold.
 2. The computer readable mediumof claim 1 wherein obtaining a set of relations comprises: obtainingcase information based on the usefulness of a constituent having thatcase; and obtaining grammatical relation information based on ausefulness of a constituent having that grammatical relation.
 3. Thecomputer readable medium of claim 2 wherein determining the relationshipcomprises: determining the usefulness of the identified constituentsbased on the case information and the grammatical relation information.4. The computer readable medium of claim 3 wherein obtaining caseinformation comprises: obtaining a hierarchy of case information; andobtaining a case hierarchy threshold based on the usefulness of aconstituent having that case.
 5. The computer readable medium of claim 4wherein obtaining grammatical relation information comprises: obtaininga hierarchy of grammatical relations; and obtaining a hierarchythreshold based on a usefulness of grammatical relations in thehierarchy in determining the relationship between the first and secondtextual inputs.